Crowdsourcing or using crowd having multiple contributors, for providing free-flowing textual annotations, for multiple tasks is a well-established trend in the industry. Examples of such tasks include, but are not limited to, translation, image captioning, content moderation, speech transcription, and so on.
In one approach, “maker-checker” style workflows are incorporated for evaluating the quality of textual annotations. However, such workflows can significantly increase latency and cost associated with the task. In another conventional approach for evaluating the quality of such contributions, automated systems have been utilized. Such automated systems provide promising results for scenarios where the work done by all the contributors is independent of each other. However, such independence of contributions cannot be guaranteed when the workers post-edit a machine input or peer-edit each other's contributions. In another improved approach, rather than enabling the contributors to provide inputs from scratch, the submissions to tasks are first processed by machines, and thereafter output of machine are post-edited or peer-edited by multiple contributors. This approach has the advantage of reducing worker effort and the cost associated with the task.
The inventors here have recognized several technical problems with such conventional systems, as explained below. When workers build upon each other's contributions, accreditation does not remain straightforward, thereby making it even more difficult to identify quality workers and quality edits. Moreover, existing approaches for determining submission and/or worker quality, do not model the dependence of the post-edit quality on prior contributions, especially in iterative peer-editing scenario.